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Abstract — We formulate a unified framework for the separa- 
tion of signals that are sparse in "morphologically" different 
redundant dictionaries. This formulation incorporates the so- 
called "analysis" and "synthesis" approaches as special cases and 
contains novel hybrid setups. We find corresponding coherence- 
based recovery guarantees for an ^i-norm based separation 
algorithm. Our results recover those reported in Studer and 
Baraniuk, ACHA, submitted, for the synthesis setting, provide 
new recovery guarantees for the analysis setting, and form a 
basis for comparing performance in the analysis and synthesis 
settings. As an aside our findings complement the D-RIP recovery 
results reported in Candes et al, ACHA, 2011, for the "analysis" 
signal recovery problem 

minimize | *x| 1 subject to ||y Ax| 2 < e 

X 

by delivering corresponding coherence-based recovery results. 



I. Introduction 

We consider the problem of splitting the signal x = xi +X2 
into its constituents xi G C d and X2 G C d — assumed 
to be sparse in "morphologically" different (redundant) dic- 
tionaries |T] — based on m linear, nonadaptive, and noisy 
measurements y = Ax + e. Here, A g C mxd , m < d, is the 
measurement matrix, assumed to be known, and e G C m is a 
noise vector, assumed to be unknown and to satisfy ||e||2 < e, 
with e known. 

Redundant dictionaries (2), |3) often lead to sparser repre- 
sentations than nonredundant ones, such as, e.g., orthonormal 
bases, and have therefore become pervasive in the sparse signal 
recovery literature p). In the context of signal separation, 
redundant dictionaries lead to an interesting dichotomy Q], 

• In the so-called "synthesis" setting, it is assumed that, 
for £ = 1,2, x e = T> e s e , where T> £ G C dxn (d < n) is 
a redundant dictionary (of full rank) and the coefficient 
vector S( G C n is sparse (or approximately sparse in the 
sense of (6)). Given the vector y G C m , the problem of 
finding the constituents xi and X2 is formalized as |7): 



(PS) 



minimize ||si||i + ||s 2 ||i 

Si,S2 

subject to ||y - A^sx + D 2 s 2 )|| 2 < £\ 



• In the so-called "analysis" setting, it is assumed that, for 
£ = 1,2, there exists a matrix G C nxd such that 
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SI/£X£ is sparse (or approximately sparse). The problem 
of recovering xi and X2 from y is formalized as (5j: 

„ . . J^ixilli + H^axalh 

'|y- A(xi +x 2 )|| 2 < e. 



minimize 

xi,x 2 

subject to 



Throughout the paper, we exclusively consider redundant dic- 
tionaries as for T>£, I — 1,2, square, the synthesis setting can 
be recovered from the analysis setting by taking \& 1 = D7 1 . 

The problems (PS) and (PA) arise in numerous applications 
including denoising [8|, super-resolution [8], inpainting (9)- 
fTT) , deblurring [11 1, and recovery of sparsely corrupted 



signals |12|. Coherence-based recovery guarantees for (PS) 
were reported in |7j. The problem (PA) was mentioned in |5). 
In the noiseless case, recovery guarantees for (PA), expressed 
in terms of a concentration inequality, are given in fl3| for 
A = l c [ and and ^2 both Parseval frames (2). 
Contributions: We consider the general problem 



minimize 

xi,x 2 

subject to ||y 



*lXi||i + ||* 2 X 2 ||l 

A1X1 - A2X2II2 < 



which encompasses (PS) and (PA). To recover (PS) from (P), 
one sets Ai = AD« and — [Id Od,n-d] T , for £ = 1,2. 
(PA) is obtained by choosing A^ = A, for £ = 1,2. Our 
main contribution is a coherence-based recovery guarantee 
for the general problem (P). This result recovers (JT] Th. 4], 
which deals with (PS), provides new recovery guarantees 
for (PA), and constitutes a basis for comparing performance 
in the analysis and synthesis settings. As an aside, it also 
complements the D-RIP recovery guarantee in (5] Th. 1.2] 
for the problem 

(P*) minimize H'S'xIli subject to ||y — Ax|| 2 < e 

X 

by delivering a corresponding coherence-based recovery guar- 
antee. Moreover, the general formulation (P) encompasses 
novel hybrid problems of the form 



minimize 

§1 ,x 2 

subject to 



llsiHi + ||*2X 2 ||l 

||y-A(D 1 si-x 2 )|| 2 <e. 



Notation: Lowercase boldface letters stand for column 
vectors and uppercase boldface letters denote matrices. The 
transpose, conjugate transpose, and Moore-Penrose inverse 
of the matrix M are designated as M T , M ff , and M^, 
respectively. The jth column of M is written [M]j, and 




(a) Original cartoon image (b) Corrupted image (c) Restored cartoon image 

Fig. 1: Image separation in the presence of Gaussian noise (SNR = 20 dB). 



the entry in the ith row and jth column of M is [Mjjj. 
We let er m i n (M) denote the smallest singular value of M, 
use I„ for the n x n identity matrix, and let Okxm be 
the k x m all zeros matrix. For matrices M and N, we 



let tu miR (M) 4 minj ||[M] 3 



t (M) 



a; min (M,N) 4 min{cj min (M),w min (N)}, and w max (M,N) 4 
max{w max (M), cj max (N)}. The fcth entry of the vector x is 
written [x]fc, and ||x||i = S/clM*-'! stands for its ^i-norm. 
We take supp fc (x) to be the set of indices corresponding 
to the k largest (in magnitude) coefficients of x. Sets are 
designated by uppercase calligraphic letters; the cardinality 
of the set S is \S\ and the complement of S (in some given 
set) is denoted by S c . For a set S of integers and n E Z, we 
let n + S = {n + p : p G S}. The n x n diagonal projection 
matrix P5 for the set S C {1, ... , n} is defined as follows: 

1, i = j and i E S 
0, otherwise, 



and we set M5 = P5M. We define crfe(x) to be the ^i-norm 
approximation error of the best fc-sparse approximation of x, 
i.e., cr^x) = ||x— X5H1 where S = supp fe (x) and X5 = P5X. 

II. Recovery Guarantees 

Coherence definitions in the sparse signal recovery litera- 
ture 1 3 J usually apply to dictionaries with normalized columns. 
Here, we will need coherence notions valid for general (un- 
normalized) dictionaries M and N, assumed, for simplicity of 
exposition, to consist of nonzero columns only. 

Definition 1 (Coherence): The coherence of the dictio- 
nary M is defined as 

|[M"M]^-| 



/t(M) = max 



,(M) 



(1) 



Definition 2 (Mutual coherence): The mutual coherence of 
the dictionaries M and N is defined as 



The main contribution of this paper is the following recov- 
ery guarantee for (P). 

Theorem 1: Let y = A1X1 + A2X2 +e with ||e||2 < £ and 
let *i e C niXpi and * 2 e C n2Xp2 be full-rank matrices. 
Let x = [xf x%} T , Ai = A(Ai*t), A2 = A(A2* 2 ). Am = 
Am(Ai*|. A 2 &1), and Amax = max{Ai, A2, Am}- Without 
loss of generality, we assume that Ai < A2- Let ki and k-i be 
nonnegative integers such that 



k\ + fc 2 < max ■ 



2(1 + A2) 



1 



A2 + 2Amax + V 7 M + Am 2 ^n 



(3) 

Then, the solution (x* , Xj ) to the convex program (P) satisfies 
||x*-x|| 2 < C E + C 1 (a kl (V l x 1 )+<j k2 (* 2 x 2 )), (4) 

where Cq,C\ > are constants that do not depend on X\ 
and X2 and where x* = [x^ T X2 T ] T . 

Note that the quantities Ai. A2> an d Am characterize the in- 
terplay between the measurement matrix A and the sparsifying 
transforms S&i and \&2- 

As a corollary to our main result, we get the following 
statement for the problem (P*) considered in j5). 

Corollary 2: Let y = Ax + e with ||e|| 2 < e and let \& G 
C nxp be a full-rank matrix. Let A; be a nonnegative integer 
such that 



k<\\l 



1 



(5) 



A(A* t ) 

Then, the solution x* to the convex program (P*) satisfies 

||x*-x|| 2 <Coe + C7 1 a fc (*x), (6) 

where Co , C\ > are constant^] that do not depend on x. 

The proofs of Theorem [T] and Corollary [2] can be found in 
the Appendix. 



(2) 



'Note that the constants Co and Ci may take on different values at each 
occuiTence. 



We conclude by noting that D-RIP recovery guarantees 
for (P*) were provided in j5). As is common in RIP-based 
recovery guarantees the restricted isometry constants are, 
in general, hard to compute. Moreover, the results in [5| 
hinge on the assumption that \I> forms a Parseval frame, i.e., 
\& H H! = I d ; a corresponding extension to general \& was 
provided in fl4) . We finally note that it does not seem possible 
to infer the coherence-based threshold <j3j from the D-RIP 
recovery guarantees in (SJ, fl4| . 

III. Numerical Results 

We analyze an image-separation problem where we remove 
a fingerprint from a cartoon image. We corrupt the 512 x 512 



greyscale cartoon image depicted in Fig. 1(a) by adding a 
fingerprinj^] and i.i.d. zero-mean Gaussian noise. 

Cartoon images are constant apart from (a small number of) 
discontinuities and are thus sparse under the finite difference 
operator A defined in fl5) . Fingerprints are sparse under the 
application of a wave atom transform, W, such as the redun- 
dancy 2 transform available in the Wave Atom toolbo^ (l6) . 
It is therefore sensible to perform separation by solving the 
problem (PA) with *i = A, \J> 2 = W, and A = I d . For our 
simulation, we use a regularized version of A and we employ 
the TFOCS solvei^from (n). 

Fig. 1(c) shows the corresponding recovered image. We can 
see that the restoration procedure gives visually satisfactory 
results. 

Appendix A 
Proofs 

For simplicity of exposition, we first present the proof of 
Corollary [2] and then describe the proof of Theorem [T] 

A. Proof of Corollary [2] 

We define the vector h = x* — x, where x* is the solution 
to (P*) and x is the vector to be recovered. We furthermore 
set S = supp fc (\&x). 

1 ) Prerequisites: Our proof relies partly on two important 
results developed earlier in |3J, |6| and summarized, for 
completeness, next. 

Lemma 3 (Cone constraint /|5^, /(S^J: The vector Sfh obeys 



\*S'Hi < ||*sh|| 1 + 2||* s .x|| 1 , 



(7) 



where S = supp^SPx). 

Proof: Since x* is the minimizer of (P*), the inequality 
||*x||i > ||*x*||i holds. Using * = * 5 + * 5 c and x* = 
x + h, we obtain 

11*5X11! + ||* S cX||i = ||* X ||i 

> ||*x*||i = ||* 5 x+ * 5 h||i + || 1 4' 5 cx + * 5 ch||i 

> ||* 5 x||i - ||*sh||i + ||* 5 -h||i - ||* 5 cx||i. 

We retrieve ([7]) by simple rearrangement of terms. ■ 



The fingerprint image is taken from h ttp : / /commons . wlkimedla . or g/ 
3 We used the WaveAtom toolbox from 
4 We used TFOCS from |http: //tfocs ■ Stanford. edu7] ~ 



http : / /www . waveatom . org/ 



Lemma 4 (Tube constraint [5], [6]): The vector Ah satis- 
fies ||Ah|| 2 < 2e. 

Proof: Since both x* and x are feasible (we recall that 
y = Ax + e with ||e||2 < e), we have the following 

||Ah|| 2 = ||A(x*-x)|| 2 

< || Ax* -y|| 2 + ||y-Ax|| 2 <2e, 

thus establishing the lemma. ■ 
2) Bounding the recovery error: We want to bound ||h||2 
from above. Since cr m i n (\I') > by assumption is assumed 
to be full-rank), it follows from the Rayleigh-Ritz theorem 1 18 
Th. 4.2.2] that 

l|h|| 2 < -4*vll* h H 2 - (9) 

^min \^ ) 

We now set Q — supp^^h). Clearly, we have for i £ Q c , 

|[*h],| < 

Using the same argument as in fl9] Th. 3.1], we obtain 



* Q ch|| 



ieQ' 



|[*h] 4 



< 



E 



l*hl 



|*eh||i 



= ll*e«h||i 



|*eh||i 



(10) 



Since Q is the set of indices of the k largest (in magnitude) 
coefficients of tyh and since Q and S both contain k elements, 
we have ||* 5 h||i < ||* Q h||i and ||* S oh||i < ||*5oh||i, 
which, combined with the cone constraint in Lemma [3] yields 



l*Q^h||^< 



* Q <=h||i < ||* e h||i +2||* 5 cx||i. 

* a h||i 



(11) 



The inequality in ( 10 1 then becomes 

l*eh|l? 



< ll*oh| 



<2||* Q h 



2||*SeX||l 

l + 2||* s .x||ii 



k 

* C h|| 2 



(12a) 



(12b) 



where ( 12a i follows from ||u||i < \Zfc||u|| 2 for fc-spars^] u 
and ( 12b i is a consequence of 2xy < x 2 + y 2 , for x, y € M. 
It now follows that 



*h| 



l*Qh||I 



*e-h||l 



\/3||* a h||2 
< v/3||* Q h|| 2 



\\*s<x\\l 
k 

|gg°xj|i 

Vk ' 



(13a) 
(13b) 



where ( 13a i is a consequence of ( |12b[ > and ( |13b[ ) results from 

yjx^+y 2 < x + y, for x, y > 0. 



Combining Q and ( |13b| i leads to 
1 



< 



5 A vector is fc-sparse if it has at most k nonzero entries. 



(14) 



3) Bounding the term \\^ Qh\\ 2 in \L4\ : In the last step of 
the proof, we bound the term || \fgh||2 in ( 14 1. To this end, we 
first bound ||A\I> t *gh|| 2 ! , with * t = ( i aPsi>)~ 1 \I> H , using 
Gersgorin's disc theorem |18] Th. 6.2.2]: 



e ma \\* Q H Z 2 < ||A*T* Qh ||^ < e„«||* c h|| 



where 9^ n 
with 



max 



1) and 6> max = 



I (15) 

M (fc - 1) 

(16) 



and w min = w min (A* t ) and cj max = w max (A* t ). 

Using Lemma |4] and ( [T5] l and following the same steps as 
in J20"| Th. 2.1] and [7 Th. 1], we arrive at the following chain 
of inequalities: 

6 mm \\* Q h\\l < HA^ehll! = (A^Qh^A^gh 
= (Ah) ff A^^gh - (A^^Qch) H A* f * Q h (17a) 

< \(Ah) a A&& a h\ + |(* Q <:h) ff (A* t ) H A* t (* a h)| 

< ||Ah|| 2 ||A*t* s h|| 2 

+ Yl I [( A* f ) ff A* f ] 1 1 [*h] i 1 1 [*h] 1 (17b) 

;n* s h|| 2 + Ai ||* s h|| 1 ||* Q ch|| 1 (He) 

J|* e h|| 2 + /i||* e h|| 1 (H^ehlli + 211*50x11!) 

(17d) 

"|* e h|| 2 + A**||*ch|l2 



< 2e^ 

< 2e. 

< 2e^ 



2/n/fe||* S cx||i||*Qh|| 



(17e) 



where ( 17a i follows from * s h = *h - * S ch and * t \E r = 
Id, ( 17b i is a consequence of the Cauchy-Schwarz inequality, 



( |17c i is obtained from \\5\ , Lemma 4] an d the definition of \x 
in (|16| , ( 17di results from (Hi, and ( 17e i comes from ||u||i < 
Vfc||u|| 2 , for /c-sparse u. 

If h ^ 0, then ||*qIi|| 2 ^ 0, since \& is assumed to 
be full-rank and Q is the set of indices of the k largest (in 
magnitude) coefficients of *h, and therefore, the inequality 
between # mm || , I'gh|| 2 and (17ei simplifies to 



(^ m -M2fc-l))||* Q h|| 2 <2 £ V^ 
This finally yields 

2e^/0^ + 2^Vk\\^/ S cx\\ 1 



2/iVfc||*s.x|| 



ll*Qh|| 2 
provided that 



< 



7 max 
,2 



- M(2fc - 1) 



/i(2fc-l) > 0. 



(18) 



4) Recovery guarantee: Using Definition [T] we get fi = 
n — ,,/,., 2 Combining (14i and (18 1, we therefore 



conclude that for 



k < 



1 



(19) 



we have 



X 2 



<C e + Ci||*s.x|| x 



C = 
d = 



£(2fe-l) ^k 



with 

2^3 

^min (^)^-'min 

cr mi „(*) ^1 
B. Proof of Theorem [JJ 

We start by transforming (P) into the equivalent problem 

(P*) minimize ||*x||i subject to ||y — Ax|| 2 < e 

X 

by amalgamating *i, * 2 and Ai, A 2 into the matrices * and 
A as follows: 




(20) 
(21) 



A = [Ai A 2 ] e C mxp 

_ _ *i nx d a (p2nx2d 
~ [Onxd *2 J 

where p — 2d in the analysis setting, p — 2n in the 
synthesis setting, and p — d + n in hybrid settings. The 
corresponding measurement vector is y = Ax + e, where 
we set x = [xj x^] 7 . 

A recovery condition for (P) could now be obtained by 
simply inserting A and \& in ( |20| , (21 1 above into Q. 
In certain cases, we can, however, get a better (i.e., less 
restrictive) threshold following ideas similar to those reported 
in (7) and detailed next. 

We define the vectors hi — x| — xi, h 2 = x 2 — x 2 , the 
sets Qi = supp fci (*ihi), Q 2 = n + supp fc2 (* 2 h 2 ), and 
h = [hf h.T} T , Q = fii U Q 2 , and set ft = fci + fc 2 . 

We furthermore let, for £ = 1,2, 

w = max \[(Ai^\) H Ai^l] itj \ 



fi m = max|[(Ai* t 1 ) H A 2 * 2 ] ij |. 

» j 

With the definitions of Qi and Q 2 , we have from ( fI3] > 
||A*t* eh ||| = ||A*t* eih ||a + ||A*tvp Q2 h||2 

+ 2(A* t * Ql h) ff A* t *Q 2 h. (22) 
The application of Gersgorin's disc theorem [18| gives 

|*o,h||? < ||A*t* ,h||*< 0^11*0, h]| 2 (23) 



7 min,l||*Qi"|l2 ^ II-"-* *Q 

|*o„h||2 < IA* 1 * 



Q 2 h||l <0 raax , 2 ||*g 2 h 



7 min.2||*Q 2 "||2 ^ II-™-* T y2 I1 ll2 ^ "max,2 || » « 2 "ll2 

,(A**]) - /^(Jfc< - 1) and 6 m 
u; max (A,*]) + fi t (kt - 1), for I = 1,2. 

In addition, the last term in |22} can be bounded as 

|(A*t* Qi h)ff A *t* Q2h | 

< ^ |[(A*t)«A*t]._ 

ieQi ,j£Q 2 

< Ai m ||* ai h||i||* Q3 h||i 

< Mm\/M52||*Q 1 h|| 2 ||* e2 h 
Z 1 ™ 7. fn.T. ul]2 



2 (24) 



[^hlilll*^ 



< r -fVhk2(\\* Ql h\\ 

<^^k- 2 \\* Qh \\l 



(25a) 
(25b) 
(25c) 

(25d) 



where ( |25a[ ) follows from the definition of p, m , (|25b"] ) results 
from ||u||i < vfc||u|| 2 , for fc-sparse u, and (25ci is a conse- 



quence of the arithmetic-mean geometric -mean inequality. 



Combining (23 I, (24 1, and (25di gives 



*oh\\i < ||A¥ T ¥oh||* < 



?min||*Q 



where min = uj^ in -f(ki,k 2 ), > 



>max||* S h||^ 



A , ,2 



^max 

+f(k 1 ,k 2 ),uj n 



W min (Ai*I, A 2 *|), Wmax — Wmax(Ai*[, A 2 *2)> and 

f(k 1 ,k 2 ) = max{//i(fci - l),fJ. 2 (k 2 - 1)} + VmVhfa- 

Using the same steps as in ( |17a[ )-( [T7e| , we get 

<Kfci,fc 2 )||* C h|| 2 < 2e,/6^+2nVk\\* S ax\\ ll 

where g(k 1: k 2 ) = - f[k x , k 2 ) - fik. 

Next, we bound g(kx, k 2 ) from below by a function of k = 
ki+k 2 . This can be done, e.g., by looking for the minimum |7 1 



g(k) = min g(ki,k-ki) 

hi : 0<ki<k 



or equivalently 



g{k)= min g(k-k 2 ,k 2 ). 

k 2 : 0<k 2 <k 



(26) 



(27) 



To find g(k) in (26 1 or in (27 1, we need to distinguish between 
two cases: 

• Case 1: fti(ki — 1) < ft 2 (k 2 — 1) 



In this case, we get 
g(k - k 2 ,k 2 ) = 



(k 2 - 1) - fi m \/k 2 (k - k 2 ) - [ik. 



A straightforward calculation reveals that the minimum of g 
is achieved at 



/'2 




resulting in 



Vi4 + i4iJ 

#0) = W mi„ - \ \^2(k - 2) + k^J^4+~^^j - Hk. 

If g(k) > 0, then we have 

||x*-x|| 2 = ||h|| 2 <C e + C 1 ||* S cx|| 1 



(28) 



where 



and 



Co 



1 



2^3 



fj min (*).9(fc) 



2fiV3k 



1 



Setting g(fc) > amounts to imposing 

2(1 + A 2 ) 



fh + 2/i max + VA2 + Am 



(29) 



where we used Definitions[T]and[2]to get a threshold depending 
on the coherence parameters only. 



• Case 2: fi 2 (k 2 — 1) < /ii(fci 
Similarly to Case 1, we get 



K*0 



2) + k 



Mi 



— /ifc. 



If <?(fc) > 0, we must have 



2(1 + Ai) 



Al + 2 Amax + + Am 



(30) 



Since Ai < A2^ by assumption, the inequality in ( 30 » 
is tighter than the one in ( |29| . We complete the proof by 
combining the thresholds in (|19[l and (29i to get (|3J. 
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